[FLINK-39251][Formats] Support Nanosecond Logical Types in Flink Avro #27770
[FLINK-39251][Formats] Support Nanosecond Logical Types in Flink Avro #27770talatuyarer wants to merge 1 commit intoapache:masterfrom
Conversation
…timestamp precision handling.
| + "with precision: " | ||
| + precision | ||
| + ", it only supports precision less than 6."); | ||
| + ", it only supports precision less than 9."); |
There was a problem hiding this comment.
It supports 9 as well I think.
|
@talatuyarer so we support with a precision of 4 but is gets treated as 6? |
MartijnVisser
left a comment
There was a problem hiding this comment.
I'd be very much in favor of splitting off the Avro dependency upgrade into its own ticket, instead of including it here.
raminqaf
left a comment
There was a problem hiding this comment.
Thanks for the contribution. I have a related PR 27757 to your changes for supporting precisions of 0-9 for the TO_TIMESTAMP_LTZ function. One thing I noticed in your changes and would like to raise awareness is:
Non-standard precisions (1, 2, 4, 5, 7, 8) are silently mapped to the next Avro-supported bucket (millis/micros/nanos) without any validation or warning. A TIMESTAMP(5) gets mapped to timestamp-micros but the converter rounds to micros, potentially losing the 5th fractional digit. Should we either (a) throw a validation error for unsupported precisions, or (b) explicitly document/warn about the truncation?
|
+1 to separate version bump from this feature. Additionally it would be good to add test cases for all precisions. Proper mapping is essential here. |
|
Seems like the |
What is the purpose of the change
Currently, the Flink Avro format support for
TIMESTAMPandTIMESTAMP_WITH_LOCAL_TIME_ZONEis capped at microsecond precision (6).Standard Apache Avro (up to 1.11) only defines logical types for
timestamp-millisandtimestamp-micros.When Flink tables utilize nanosecond precision (
TIMESTAMP(9)), there is no native Avro logical type mapping, leading to either fallback behaviour (treating as rawBIGINT) or unsupported type exceptions during automatic schema conversion.This pull request adds support for nanosecond-precision logical types (
timestamp-nanosandlocal-timestamp-nanos) in the Flink Avro format. It also upgrades Avro to version 1.12.1 and enables decimal logical types handling for improved compatibility and numeric fidelity.Specifically:
TIMESTAMP(9)andTIMESTAMP_WITH_LOCAL_TIME_ZONE(9)back-and-forth with Avro'stimestamp-nanosandlocal-timestamp-nanoslogical types.Brief change log
AvroToRowDataConverters.java:createTimestampConverter(int precision)that handles millisecond, microsecond, and nanosecond cases intoTimestampData.RowDataToAvroConverters.java:TIMESTAMP_WITHOUT_TIME_ZONEandTIMESTAMP_WITH_LOCAL_TIME_ZONEto respect higher precision values up to 9 (nanoseconds) when packing epoch values.AvroSchemaConverter.java:LogicalTypes.timestampNanos()andLogicalTypes.localTimestampNanos()schemas into equivalent Flink SQL styles (TIMESTAMP(9)/TIMESTAMP_WITH_LOCAL_TIME_ZONE(9)).pom.xml:Verifying this change
This change added tests and can be verified as follows:
AvroSchemaConverterTest.java(e.g., asserting fields liketype_timestamp_nanos,type_local_timestamp_nanosmap correctly).AvroOutputFormatTest,AvroTypeExtractionTest) to ensure backwards compatibility with upgraded baseline versions.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation